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Abstract 

Using the game-theoretic framework for probability, Vovk and Shafer [10] have shown that it 
is always possible, using randomization, to make sequential probability forecasts that pass any 
countable set of well-behaved statistical tests. This result generalizes work by other authors, 
who consider only tests of calbration. 

We complement this result with a lower bound. We show that Vovk and Shafer's result is 
valid only when the forecasts are computed with unrestrictedly increasing degree of accuracy. 

When some level of discreteness is fixed, we present a game-theoretic generalization of Oakes' 
example for randomized forecasting that is a test failing any given method of deferministic 
forecasting; originally, this example was presented for deterministic calibration. 
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1. Introduction 

Using the game-theoretic framework for probability [9], Vovk and Shafer have shown 
in [10] that it is always possible, using randomization, to make sequential probability fore- 
casts that pass any countable set of well-behaved statistical tests. This result generalizes 
work by other authors, among them are Foster and Vohra [4], Kakade and Foster [5], 
Lehrer [6], Sandrony et al. [8], who consider only tests of calibration. 

We complement this result with a lower bound. We show that Vovk and Shafer's result 
is valid only when the forecasts are computed with unrestrictedly increasing degree of 
accuracy. When some level of discreteness is fixed, we present a game-theoretic version 
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of Oakes' example for randomized forecasting that is a test failing any given method of 
deterministic forecasting; originally, this example was presented for deterministic cali- 
bration. To formulate this example, we use the forecasting game presented by Vovk and 
Shafer [10], namely Binary Forecasting Game II. 

We discuss details of the randomized forecasting algorithms in Section 2. 

The Shafer and Vovk's [9] game-theoretic framework is considered in Section 3. We 
present in this section the original Vovk and Shafer's [10] result on universal randomized 
forecasting and prove our result which gives the limits for such forecasting - a game- 
theoretic version of the Oakes' example for randomized forecasting. 

2. Background 

The research discussed in this paper was started from a notion of calibration, originated 
by Dawid [1,2]. A test of calibration checks whether the observed empirical frequencies 
of state occurrences converge to their forecaster probabilities. 

Let I(p) denote the indicator function of a subinterval / C [0, 1], i.e., I(p) = 1 if p G I, 
and I(p) — 0, otherwise. An infinite sequence of forecasts pi,P2, ■ ■ ■ is well-calibrated for 
an infinite sequence of outcomes UJ1OJ2 ... if for the characteristic function I(p) of any 
subinterval of [0, 1] the calibration error tends to zero, i.e., 

E?=i-t(p«)(^«-p«) n m 

as the denominator of the relation (1) tends to infinity. Here, I(pi) determines some 
"selection rule" which defines indices i where we compute the deviation between forecasts 
Pi and outcomes Wj. 

The main problem of sequential forecasting is to define a universal forecasting algo- 
rithm which computes forecasts p n given past observations u>\, . . . , Lu n ~i for each n. This 
universal prediction algorithm should be well-calibrated for each infinite sequence of out- 
comes. Oakes [7] proposed arguments (see Dawid [3] for a different proof) that no such 
algorithm can be well-calibrated for all possible sequences: any forecasting algorithm 
cannot be calibrated for the sequence u> = u)\lo2 . . ., where 




1 if pi < 0.5 
otherwise 



and pi are forecasts computed by the algorithm given oj\, . . . , i = 1,2, . . .. The 

corresponding intervals are Iq = [0,0.5) and I\ = [0.5,1]. It is easy to see that the 
condition (1) of calibration fails for this ui, where / = I or / = I 1 . 

Foster and Vohra [4] show that calibration is almost surely guaranteed with a random- 
izing forecasting rule, i.e., where the forecasts are chosen using internal randomization. 
Kakade and Foster [5] noticed that some calibration results require very little randomiza- 
tion. They defined "an almost deterministic" randomized rounding universal forecasting 
algorithm /: for any sequence of outcomes u>iu>2 . . ., an observer can only randomly round 
the deterministic forecast up to A in order to calibrate with the internal probability 1 
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5^/(Pi)(wi-Pi) <A 



(2) 



i=l 



where A is the calibration error, I{p) is the indicator function of an arbitrary subinterval 



This approach was further developed by, among others, Lehrer [6], Sandrony et al. [8]. 
These papers were only concerned with asymptotic calibration. Non-asymptotic version 
of randomized forecasting was proposed by Vovk and Shafcr [10] and by Vovk et al. [11]. 
They based on the game-theoretic framework of Shafer and Vovk [9] . 

Let V{0, 1} be the set of all measures on the two-element set {0, 1}. Any measure from 
V{0, 1} is represented by a number p G [0, 1] - the probability of {1}. We consider also 
the set of all measures on ^{0, 1}. Let V[0, 1] be the set of all probability measures on 
the unit interval [0, 1] supplied with the standard Borel er-ficld T. 

Randomizing forecasting is defined as follows. For each n, a forecaster given a binary 
sequence of past outcomes w\ . . . to n -i (and a sequence of past forecasts p\, . . . ,p n -i) 
outputs a probability distribution P n 6 P[0, 1]. The forecasts p n of the the future event 
u> n = 1 are distributed according to this probability distribution. 

Assume for each n, the probability distribution P n is concentrated on a finite subset 
D n of [0,1], say, D n = {p 

n,ii ■ ■ ■ iPn.nin}- The number A — liminf A n , where 



is called the level of discreteness of the corresponding forecasting scheme on the sequence 

UJ\UJ2 ■ ■ ■■ 

In general case D n is a predictable random variable, i.e., measurable with respect to 
the cr-field JT™ -1 , depending on uj\ . . . u n -\. 

A typical example is the uniform rounding: for each n, rational points p n ^ divide the 
unit interval into equal parts of size < A < 1 and P n is concentrated on these points. 
In this case the level of discreteness on arbitrary sequence W\0J2 ■ ■ ■ equals A. 

3. Game-theoretic randomized forecasting 

Shafer and Vovk [9] proposed a game-theoretic framework for probability theory. In 
Vovk and Shafer [10] they used this framework to demonstrate the possibility of good 
probability forecasting in a general setting. This result generalizes the previous work of 
many authors. 

Vovk and Shafer presented a game between Reality, Forecaster and Skeptic. In this 
game, Forecaster faces Skeptic whose strategy is revealed in advance, and he is allowed 
to use a degree of randomization to conceal each of his probability forecasts until the 
corresponding outcome has been announced. Their main result says that Forecaster can 
keep Skeptic from becoming infinitely reach. Intuitively, this means that the outcomes 
determined by Reality look random with respect to probability forecasts. This result is 
a consequence of the von Neumann's minimax theorem. 

Vovk and Shafer [10] consider a perfect-information game of randomized forecasting - 
Binary Forecasting Game //between three players - Forecaster, Skeptic, Reality, Random 
Number Generator described by the following protocol: 



of [0,1]. 



n 



oc 



A„ = inf{|p nil - p nJ \ : i ^ j} 
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Let /Co = 1 and JF = 1. 
FOR n = 1,2,... 

Skeptic announces 5„ : [0, 1] — > 1Z. 

Forecaster announces a probability distribution P n £ V[0, 1]. 
Reality announces w„ E {0, 1}. 

Forecaster announces /„ : [0, 1] — > TZ such that / f n (p)Pn(dp) < 0. 
Random Number Generator announces p n £ [0, 1]. 
Skeptic updates his capital fC n = K n -\ + S n (p n )(u> n — p n ). 
Forecaster updates his capital T n = T n -\ + f n {Pn)- 
ENDFOR 

Restriction on Skeptic: Skeptic must choose the S n so that his capital K. n is nonnegative 
for all n no matter how the other players move. 

Restriction on Forecaster: Forecaster must choose the P n and /„ so that his capital 
T n is nonnegative for all n no matter how the other players move. 

Vovk and Shafcr [10] showed that Forecaster has a winning strategy in the Forecasting 
Game II, where Forecaster wins if either (i) his capital T n is unbounded or (ii) Skeptic's 
capital IC n stays bounded; otherwise the other players win. 

Theorem 1 Forecaster has a winning strategy in Binary Forecasting Game II. 

Sketch of the proof. For completeness of the presentation, we reproduce the proof from [10] . 

The proof is based on von Neumann's minimax theorem. 

At first, at any round n of Binary Forecasting Game II, a simple auxiliary game between 
Realty and Forecaster is considered: Forecaster chooses p n £ [0,1], Realty chooses w„ G 
{0, 1}. Forecaster losses (and Realty gains) S(p n )(u) n — p n ). 

For any mixed strategy of Realty Q n £ V{0, 1}, let Forecaster's strategy be p n = Q{1}. 
So, the Realty's expected gain is S(p„)(l - Q{1})Q{1} + S (p n )(0 - Q{1})(1 - Q{1}) = 0. 

In order to apply von Neumann's minimax theorem, which requires that move space be 
finite, we replace Forecaster move space [0, 1] with a finite subset of [0, 1] dense enough 
that the value of the game is smaller than some arbitrary small positive number A 
(depending on n). This is possible, since | S n (p) \ < JC n -i < 2" _1 . 2 The minimax theorem 
asserts that Forecaster has a mixed strategy P £ V[0, 1] such that 



for both cj n = and oj n = 1. 

Let Ea be the subset of V[0, 1] consisting all probability measures P satisfying (3) for 
u n = and uu n = 1. Endowed with the weak topology, V[0, 1] is compact. Since each 
Ea is closed, fli^A; ^ ®> where Aj, i = 1, 2, . . ., is some decreasing to sequence of real 
numbers. So there exists P n £ V[0, 1] such that 



for both uj n = and uj n = 1. 

In Binary Forecasting Game II, consider the strategy for Forecaster that uses at any 
round n the probability distribution P n just defined and uses as his second move the 



2 Skeptic must choose S n (p) such that /Cn > for all n no matter the other players move. 




(3) 




'n 
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function /„ defined f n (p) — S n {p)(Lo n — p)- Then T n — JC n for all n. So either Skeptic's 
capital will stay bounded or Forecaster's capital will be unbounded. A 

Vovk et al. [11] (see also [10]) also showed that Skeptic can present a strategy S n (p) 
such that the winning strategy of Forecaster existing by Theorem 1 announces forecasts 
pi,P2, ■ ■ ■ which are well-calibrated for an arbitrary sequence LO1LO2 • • ■ of outcomes. 

In that follows we consider some version of the Oakes' example in the game-theoretic 
framework. A different version of this result is given in [12]. 

We consider some modification of Binary Forecasting Game II in which Skeptic (but 
not Forecaster) announces /„ : [0,1] — > 7Z. This means that Skeptic defines the test of 
randomness he needs. 

Also, at each step n, Skeptic divide his capital into two accounts: IC n = Q n + T n \ he 
uses the capital T n to force Random Number Generator to generate random numbers 
which pass the test /„. 

Let /C = 2. 
FOR n = 1,2,... 

Skeptic announces S n : [0, 1] — > 7Z. 

Forecaster announces a probability distribution P n E V[0, 1]. 
Reality announces u> n 6 {0, 1}. 

Skeptic announces /„ : [0, 1] — > 1Z such that J f n (p)P n (dp) < 0. 

Random Number Generator announces p n £ [0, 1]. 

Skeptic updates his capital JC n = K, n -i + S n (p n )(uj n - p n ) + f n (p n ). 

ENDFOR 

We divide the Skeptic's capital into two parts: 
= Qn + F n f° r an n , where 
Qo = 1 and T = I. 

Qn = Qn-i + S„(p n )(u>„ -p n ) and 

•F; n 3~ n—1 In iPn) ■ 

Restriction on Skeptic: Skeptic must choose the S n and /„ so that his capital K n is 
nonnegative for all n no matter how the other players move. 

Actually, Skeptic will choose the S n and /„ so that both of his capitals Q n and T n arc 
nonnegative for all n no matter how the other players move. 

We prove that when Forecaster uses finite subsets of [0, 1] for randomization Realty 
and Skeptic can defeat Forecaster (and Random Number Generator) in this forecasting 
game, where Realty and Skeptic win if Skeptic's capital K n is unbounded; otherwise 
Forecaster and Random Number Generator win. 

Theorem 2 Assume Forecaster's uses a randomized strategy with a positive level of 
discreteness on each infinite sequence u). Then Realty and Skeptic win in the modified 
Binary Forecasting Game II. 

Proof. Define a strategy for Realty: at any step n Realty announces an outcome 



We follow Shafer and Vovk's [9] method of defining the defensive strategy for Skeptic. 
Let e k = 2- k , k = 1, 2, . . .. We define recursively by n: Q s ' k = 1, S s ' k (p) = 0, s = 1, 2, 
and for n > 1 





1 otherwise. 
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# fc (p) = -e fc e^(p>0.5), (4) 
# fc (p) = efc <#*i£(p<0.5), (5) 
where £(true) — 1, £(false) = 0, and for n > 1 

Q^ = Q^ 1 + ^ fe (p„)K-p„)), (6) 
Q^ = Q^ 1 + 5^ fe (p„)K-p„)). (7) 

We combine S„ k (p) and S^' fe (p) in the Skeptic's strategy 5„(p) = ^(S„(p) + S„(p)), 
where 

oo 

Si(p) = 5># fc (p) 
fe=i 

and 

oo 

s£(p) = £ c *# fc (p)- 
fe=i 

It can be proved by the mathematical induction on n that < Q]i k < 2™ and |S^ fc (p)| < 
2«-i f or j = i ; 2 and for all fc, p and n. Then these sums are finite for each n and p. 

By (6)-(7) the Skeptic's capital Q„ at step n, when he follows the strategy S n (p), 
equals 



2 fc =r 

Define for each n the function g n (p) — (2£(p < 0.5) — l)(co n — p). Let Ep n (g n ) = 
1 9n(p)P n (dp). 

Recall that Forecaster uses some randomized strategy P n , n = 1, 2, 

We define recursively by n: T§ = 1, g^(p) — 0, and for n > 1 

<£(p) - -e^.Mp) - E Pn (g n )), (8) 
where = 2~ fe , and for n > 1 

^=^-l + <£(P») (9) 

By definition for any h and n, 

n 

^ = n( i - e ^ 3 (p,) - E Pj ( 9j ))y (io) 

By (10) < ^* < 2™ for all n and fc. 
Finally, Skeptic defines at step n 

oo 

fe=l 

By definition j f n (p)P n (dp) < 0. 

By (10) the Skeptic's capital T n at step n, when he follows the strategy f n (p), equals 



fc=l 
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Also, T n > for all n. 

Suppose that sup.F„ = C < oo, where C > 0. Then supJT^ < for each fc. 

We have for each fc, 

n 

ln^ > -effete) - - «Cfc- 

Here we use the inequality ln(l + r) > r — r 2 for all r < i. 
Since ,F n is bounded by C > 0, we have for any fc 

\ fx^ - ^ fe)) > - inC + in(£fc) - efe * - 2efe ( u ) 

for all sufficiently large n. 
Define two variables 

n 

#n,l = E^' > °- 5 )( W J ~^')' 
n 

l?n,2 = ^Zfrj < 0.5)(Wj -Pj)- 

By definition of <7j, 

n 

i=i 

For technical reason define gij(p) = £(p > 0.5) (wj — p) and g2,j(p) = £(p < 0.5) (u>j — p). 
Then ^(p) = ff 2 ,j(p) - 5i,j(p)- 

Assume for any n the probability distribution P n is concentrated on a finite set 

{Pn,l; ■ • ■ 7 Pn,m n } • 

For technical reason, if necessary, we add and 1 to the support set of P n and set their 
probabilities to be 0. Denote p~ = max{p„. t : p n ^ t < 0.5} and p+ = min{p„ it : p n ^ t > 
0.5}. 

By definition ui n , p+ and p~ are predictable and p+ — p~ > A for all n, where A > 0. 
We have 

n 

Y, e pM,o) < E p i& > o-5}(-p+) + E > °- 5 Ki -p/) < 

j — 1 cjj— ^j — 1 

n n 

-0.5 X^K- = 0)p+ + 0.5^?( Wj = 1)(1 -p+). (12) 

n 

E^-(^) > E p i(P ^ °- 5 >(-p7) + E p oiP ^ 0.5}(1 -p7) > 

j — 1 cjj— Wj=l 

n n 

-0-5 E^K' = %7 + 0.5^^ = i)(i-P7)- (13) 



Subtracting (12) from (13), we obtain 

n n n 

E (9j) = E E r> (92 j) - E '■■<■ (91 j) > 0-5 An. 

i=i i=i j=i 

Using (11), we obtain for all sufficiently large n 
1 1 ™ 1 ™ 

-(0n,2 " = - E 9j(Pj) > ~ E Ep i (9i) ~ 2c * ^ °' 5A - 2e *- ( U ) 

Now we compute a lower bound of Skeptic's capital. 
We have from the definition (4)-(5) and (15)-(16). 

n 

Q 1 / = II^ 1 - e ^fe > 0-5)(^- - ft)), (15) 

n 

Qn k = II^ 1 + e ^fe ^ 0.5)K -p,)). (16) 

j=l 

By (15) and (16), for i = 1,2, < Q^ fe < 2" for all n no matter how the other players 
move. 

By (15)-(16) at step n 

lnQ^>-eA,i-^, (17) 
lnQn k > e k #n,2 - e 2 k n. (18) 

The inequalities (17), (18) and (14) imply 

limsup ln g " + ln g " > 0.5e fc A - 2e 2 k > 2e\ (19) 

for all sufficiently large n, where e k < |A. 

From this, we obtain that for i = 1 or for i = 2, 

hm sup > e fe 

for all sufficiently large n. 

Hence, we obtain for the total capital of Skeptic K n = Q n + T n 

lim sup K n = oo 

n— >oo 

no matter how Forecaster moves if Realty uses her strategy defined above. 

We obtain also a lower bound of calibration error for Binary Forecasting Game II. 
Corollary 1 Assume Forecaster's uses a randomized strategy with a positive level of 
discreteness on each infinite sequence u). Then Realty can announce an infinite binary 
sequence wiw 2 . . . such that 

limsup -i?„ ;i > 0.25A (20) 

n— >oo 71 



for i — 1 or for i = 2. 

This inequality immediately follows from (14). 
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